[otbn] Add synchronization interface between OTBN and Ibex by etterli · Pull Request #30660 · lowRISC/opentitan

etterli · 2026-07-03T15:14:25Z

This PR contains the specification and implementation of a synchronization interface between Ibex and OTBN. OTBN can execute a WFI instruction which will stall the execution until Ibex writes a RESUME command to OTBN's CMD register.

The simulator adaptation, a top level test and some simple DV sequences follow in separate PRs (work is done).

The WFI instruction can be enabled at runtime via a new bit wfi_enabled in the CTRL register.
I deliberately haven't added a REGWEN to the CTRL register. I think we don't need this (see below) and I also don't know whether the already existing bit software_errs_fatal can be locked for all applications (I don't know how CL uses/sets this bit as of now).

IMHO, we don't need to lock this CTRL register because:

Any adversary enabling this cannot gain any advantage when a SW runs which does not make use of the WFI instruction. There is no effect to programs not making use of WFI.
And if a SW runs which uses WFI, then I don't see a problem when someone disables the WFI instruction. In this case the instruction is decoded as illegal.

In addition, any adversary could also directly attack the decoder of OTBN to fault this configuration. Due to these reasons I don't see the need to add a REGWEN for CTRL.

etterli · 2026-07-14T13:55:16Z

This is now ready to review. Please read the updated PR description.

rswarbrick

Some detailed comments about the spec side of things, and a couple of notes about otbn.sv. I haven't read the rest of the design changes.

If there has already been a group decision that "WFI" is appropriate, I guess the review comments about the name are too late. That would be rather sad though.

rswarbrick · 2026-07-15T09:27:30Z

-        // it ignores the write and the test would fail.
+        // Don't write this register in the automated CSR tests, because those tests are not aware
+        // whether OTBN is accepting commands or not. If OTBN is not accepting commands, it ignores
+        //the write and the test would fail.


Nit: Missing space.

rswarbrick · 2026-07-15T09:29:38Z

+When OTBN is [paused](#operational-states) by a {{#otbn-insn-ref WFI}} instruction, only the DMEM is unlocked and accessible, while the IMEM remains locked.
+Note that pausing is only possible when the {{#otbn-insn-ref WFI}} instruction is enabled in the [`CTRL`](registers.md#ctrl) register.
+For any memory that is not accessible in the current state, reads return zero and writes have no effect.
+Furthermore, an access to such an inaccessible memory will cause OTBN to generate a fatal error with code `ILLEGAL_BUS_ACCESS`, unless OTBN is locked.


Is it possible for DMEM/IMEM to be locked when OTBN is locked?

The old text was "a memory access when OTBN is neither idle nor locked will cause OTBN to generate a fatal error". So the memories can be accessed either in the idle or locked state. See the assignment imem_access_core and dmem_access_core in otbn.sv. Both the previous and new implementation allow this access. However, when OTBN is locked any read of the memories will return 0. See for example the imem_rdata_bus_en_d signal at otbn.sv:742.

If your question was if this can be changed so that the memories are locked when OTBN is locked: Yes would be possible. But as only 0 is returned I think there is no need to change the design.

rswarbrick · 2026-07-15T09:32:56Z

+  - A {{#otbn-insn-ref WFI}} instruction is executed (if enabled).
+    In this case, the execution is paused and the DMEM is unlocked for the host.
+    The execution must be resumed by the host by issuing the `RESUME` command.
+    There can be multiple pauses per execution and each {{#otbn-insn-ref WFI}} instruction issues the `done` interrupt ([`INTR_STATE.done`](registers.md#intr_state)).


At the moment, the "Wait For Interrupt" instruction triggers an interrupt and then waits for a (non-interrupt) ack from the host processor.

Has the name already been assigned somewhere? If not, I strongly advise renaming the instruction to something like PAUSE

I chose the name WFI because this pausing feature is actually matching the WFI instruction defined by the RISCV specs (see privileged specs, chapter 3.3.3). I think it is a suitable name because:

I agree from a system view the WFI generates an interrupt which seems counter-intuitive. But looking from an OTBN SW perspective, the WFI instruction just pauses the execution until an interrupt happens. In this case the interrupt source is a change in OTBN's CTRL register. Which I think is a perfectly valid interrupt source.

That the 'done' interrupt is fired when a WFI instruction is hit is also a common and in my opinion valid behaviour. The RISCV spec says:

Execution of the WFI instruction can also be used to inform the hardware platform that suitable interrupts should preferentially be routed to this hart.

In our case the hardware platform is the Ibex / receiver of the 'done' interrupt. So I think firing the done interrupt is a valid action upon reaching the WFI instruction.

In addition, a similar behaviour can be found multi core systems / clusters. There are synchronization / fence instructions which pause a core until all other cores also reached this instruction. And each core then fires an 'interrupt' which the cluster controller registers. Once all cores are waiting this cluster controller then sends a signal to all cores to resume the execution.

And another idea why it should be named WFI is that I had the idea that in future a separate CSR can be used to mask out / enable certain interrupts for WFI if OTBN will ever feature interrupts (maybe from another accelerator/interface). So the WFI instruction then could either wait for a CTRL change or any other interrupt.

rswarbrick · 2026-07-15T09:36:09Z

  interrupt_list: [
    { name: "done"
-      desc: "OTBN has completed the operation."
+      desc: "OTBN has completed the operation or encountered a WFI instruction."


How difficult would it be to rename the interrupt? "DONE" is definitely incorrect now.

"STOPPED"?

I agree the naming is not perfectly describing what it does.

Renaming it on the RTL side is okaish from an effort perspective. On the SW side we would have to change:

otbn_testutils_wait_for_done in otbn_testutils.h. Used 16 times.

otbn_busy_wait_for_done in cryptolib drivers. Used 69 times.

sc_otbn_busy_wait_for_done in silicon creator drivers. Used 7 times.
But I think there is a considerable effort to clean up the simulator and DV side.

What we were thinking as alternative was to introduce a new interrupt. But we decided against it because it is kind of not necessary to have more than 1 interrupt. Any host starting an OTBN SW which will pause knows that the first interrupt is the pause.

And a silly reason to keep the name is that one can see it as indicating that a part of the program is done when the interrupt fires ;)

rswarbrick · 2026-07-15T09:42:44Z

  logic is_not_running_d, is_not_running_q;
  logic otbn_dmem_scramble_key_req_busy, otbn_imem_scramble_key_req_busy;

+  // While paused the clock must not be gated to be able to detect the RESUME command and react to


Grammar nit: This needs slightly reordering to avoid it being ambiguous. It looks a bit like it says gating would allow it to detect the RESUME command...

More importantly, the comment looks like it's describing the whole statement, but is actually just describing the second line.

I removed this comment and described the topic in the already existing commend just above.

rswarbrick · 2026-07-15T09:43:40Z


  // Interrupts ================================================================

+  // Fires when OTBN gets paused or ends execution.


As a status interrupt, I'd avoid "fires" (because that sounds more like a particular point in time)

Rephrased to "Active when...".

rswarbrick · 2026-07-15T09:45:51Z

+  // Mux core and bus access into IMEM.
+  // While paused by a WFI insn the IMEM stays locked.


I'd probably explain this signal better. How about this?

// IMEM access is granted to the core (not the bus) if the core is executing, being started, or is // paused by a WFI instruction.

rswarbrick · 2026-07-15T09:49:20Z


-  // Mux core and bus access into dmem
+  // Mux core and bus access into dmem. While paused, busy_execute_q is low so the DMEM is
+  // unlocked to the bus.


Nit: Probably "accessible" is better here.

Here, access is unlocked (although that sounds rather clunky) but the DMEM is either accessible (in that case) or not (when it is locked).

What a silly language!

True, thanks. Changed it.

rswarbrick · 2026-07-15T09:50:35Z

+    end else if (busy_execute_q || status_q == StatusPaused) begin
      // OTBN can command a secure wipe of IMEM and DMEM. This occurs when OTBN encounters a fatal
-      // error.
+      // error. These can also occur when paused.


I agree that this is true, but does it actually need saying explicitly?

No, removed the new sentence.

etterli

Thanks @rswarbrick for the comments! I answered to the three main questions / issues and fixed the other comments.

etterli · 2026-07-15T12:55:09Z

+When OTBN is [paused](#operational-states) by a {{#otbn-insn-ref WFI}} instruction, only the DMEM is unlocked and accessible, while the IMEM remains locked.
+Note that pausing is only possible when the {{#otbn-insn-ref WFI}} instruction is enabled in the [`CTRL`](registers.md#ctrl) register.
+For any memory that is not accessible in the current state, reads return zero and writes have no effect.
+Furthermore, an access to such an inaccessible memory will cause OTBN to generate a fatal error with code `ILLEGAL_BUS_ACCESS`, unless OTBN is locked.


The old text was "a memory access when OTBN is neither idle nor locked will cause OTBN to generate a fatal error". So the memories can be accessed either in the idle or locked state. See the assignment imem_access_core and dmem_access_core in otbn.sv. Both the previous and new implementation allow this access. However, when OTBN is locked any read of the memories will return 0. See for example the imem_rdata_bus_en_d signal at otbn.sv:742.

If your question was if this can be changed so that the memories are locked when OTBN is locked: Yes would be possible. But as only 0 is returned I think there is no need to change the design.

etterli · 2026-07-15T13:20:56Z

+  - A {{#otbn-insn-ref WFI}} instruction is executed (if enabled).
+    In this case, the execution is paused and the DMEM is unlocked for the host.
+    The execution must be resumed by the host by issuing the `RESUME` command.
+    There can be multiple pauses per execution and each {{#otbn-insn-ref WFI}} instruction issues the `done` interrupt ([`INTR_STATE.done`](registers.md#intr_state)).


I chose the name WFI because this pausing feature is actually matching the WFI instruction defined by the RISCV specs (see privileged specs, chapter 3.3.3). I think it is a suitable name because:

I agree from a system view the WFI generates an interrupt which seems counter-intuitive. But looking from an OTBN SW perspective, the WFI instruction just pauses the execution until an interrupt happens. In this case the interrupt source is a change in OTBN's CTRL register. Which I think is a perfectly valid interrupt source.

That the 'done' interrupt is fired when a WFI instruction is hit is also a common and in my opinion valid behaviour. The RISCV spec says:

Execution of the WFI instruction can also be used to inform the hardware platform that suitable interrupts should preferentially be routed to this hart.

In our case the hardware platform is the Ibex / receiver of the 'done' interrupt. So I think firing the done interrupt is a valid action upon reaching the WFI instruction.

In addition, a similar behaviour can be found multi core systems / clusters. There are synchronization / fence instructions which pause a core until all other cores also reached this instruction. And each core then fires an 'interrupt' which the cluster controller registers. Once all cores are waiting this cluster controller then sends a signal to all cores to resume the execution.

And another idea why it should be named WFI is that I had the idea that in future a separate CSR can be used to mask out / enable certain interrupts for WFI if OTBN will ever feature interrupts (maybe from another accelerator/interface). So the WFI instruction then could either wait for a CTRL change or any other interrupt.

etterli · 2026-07-15T13:38:15Z

  interrupt_list: [
    { name: "done"
-      desc: "OTBN has completed the operation."
+      desc: "OTBN has completed the operation or encountered a WFI instruction."


I agree the naming is not perfectly describing what it does.

Renaming it on the RTL side is okaish from an effort perspective. On the SW side we would have to change:

otbn_testutils_wait_for_done in otbn_testutils.h. Used 16 times.

otbn_busy_wait_for_done in cryptolib drivers. Used 69 times.

sc_otbn_busy_wait_for_done in silicon creator drivers. Used 7 times.
But I think there is a considerable effort to clean up the simulator and DV side.

What we were thinking as alternative was to introduce a new interrupt. But we decided against it because it is kind of not necessary to have more than 1 interrupt. Any host starting an OTBN SW which will pause knows that the first interrupt is the pause.

And a silly reason to keep the name is that one can see it as indicating that a part of the program is done when the interrupt fires ;)

etterli · 2026-07-15T13:41:53Z

  logic is_not_running_d, is_not_running_q;
  logic otbn_dmem_scramble_key_req_busy, otbn_imem_scramble_key_req_busy;

+  // While paused the clock must not be gated to be able to detect the RESUME command and react to


I removed this comment and described the topic in the already existing commend just above.

etterli · 2026-07-15T13:43:30Z


  // Interrupts ================================================================

+  // Fires when OTBN gets paused or ends execution.


Rephrased to "Active when...".

etterli · 2026-07-15T13:44:15Z

+  // Mux core and bus access into IMEM.
+  // While paused by a WFI insn the IMEM stays locked.


etterli · 2026-07-15T13:44:51Z


-  // Mux core and bus access into dmem
+  // Mux core and bus access into dmem. While paused, busy_execute_q is low so the DMEM is
+  // unlocked to the bus.


True, thanks. Changed it.

etterli · 2026-07-15T13:45:24Z

+    end else if (busy_execute_q || status_q == StatusPaused) begin
      // OTBN can command a secure wipe of IMEM and DMEM. This occurs when OTBN encounters a fatal
-      // error.
+      // error. These can also occur when paused.


No, removed the new sentence.

h-filali · 2026-07-15T14:33:08Z

  // Interrupts ================================================================

+  // Active when OTBN gets paused or ends execution.
  assign done = is_busy_status(status_q) & ~is_busy_status(status_d) & init_sec_wipe_done_q;


I'm wondering if there is any error that would send a done pulse in the running state but now doesn't in the paused state.

In other words, is there any way that if we run into an issue that requires a transition into the locked state while we are in the paused state? And if so what happens if we don't see the done pulse there?

Good question!

Right now done is pulsed when the locked stated is entered from any busy status (BusyExecute or BusySecWipe*). But done is not pulsed on a direct StatusPaused to StatusLocked edge (because is_busy_status(StatusPaused) is false).
In practice the "fatal error while paused" path is safe because it detours through StatusBusySecWipeInt before locking.

However there is the possibility that we have to transition from StatusPaused directly into StatusLocked. This can happen under fault attack. I made it that this transition also issues the done interrupt. Thanks for spotting this!

The wipe command was selected from the cmd_e enum with the restriction it may not be the execute command. However, it is planned to add more commands which are not a valid wipe command. This restricts the wipe command randomization to wipe commands only. Signed-off-by: Pascal Etterli <pascal.etterli@lowrisc.org>

etterli · 2026-07-16T14:16:12Z

rebased on master to pick up the FPGA test split (which hopefully avoids the timeouts)

vogelpi

Thanks @etterli , this is very nice work!

I have some questions but the whole thing looks very clean and nice, well done!

vogelpi · 2026-07-16T15:55:54Z

+  // Interrupt is set when OTBN is no longer busy. This is the case when it gets paused or ends
+  // execution. The done interrupt is also set when a direct paused to locked transition occurs.
+  // This can happen under some faults where the secure wipe is skipped. These faults raise a fatal
+  // alert but for consistency we still set the interrupt. The faults are:
+  // - An invalid MuBi on lc_escalate_en_i
+  // - Spurious URND reseed ACK


This corner case should also be mentioned in the documentation of the done interrupt. Alternatively, one could discuss to alter the behavior around the corner case to simplify DV.

This isn't actually a corner case, it is just to keep the behaviour the same as before. Without the WFI feature, when OTBN transitions from busy/executing to locked the DONE interrupt is still raised. See @h-filali question here. This new change is just to make the behaviour consistent and ensures that the DONE interrupt is also raised when transitioning from paused to locked.

I agree that the current description of the DONE interrupt (old and new) just says that it is raised when leaving a busy state. I added that it is also raised when OTBN locks itself. See updated otbn.hjson.

vogelpi · 2026-07-16T16:05:09Z

+  // wfi_pending signal has only effect if we haven't already received the resume command.
+  assign busy_execute_d = wfi_resume_d |
+                          ((busy_execute_q | start_d) &
+                           ~(done_core | (wfi_pending & ~wfi_resume_q)));


IIUC, you need this last term here because the RESUME command will remain in the register and you just want a pulse to move forward.

Question: Would it make anything simpler instead of asking Ibex to write RESUME, we would just wait for Ibex to write again EXECUTE?

No I need the & ~wfi_resume_q part because wfi_pending is driven by the WFI instruction which is still executing when we resume, i.e., wfi_resume_q = 1. As long as the wfi instruction is executing, wfi_pending = 1 and thus busy_execute_d would be raised one cycle too late.

vogelpi · 2026-07-16T16:08:44Z

+  end
+  assign wfi_pending_o = wfi_pending_q;
+
+  assign stall = mem_stall | ispr_stall | rf_indirect_stall | mac_bignum_stall | wfi_stall;


Can you please cross check that the stall signal here isn't factored into the critical path? Because the factors in the unregistered wfi_stall and I remember you cleaned this up in the past before adding the vector instructions.

Hm I don't think the new wfi_stall signal extends any path stall factors in at all. The wfi_stall is derived from:

wfi_resume_i: This originates directly from the flop wfi_resume_q in otbn.sv.

insn_valid_i: Is already factored in other stall sources like mac_bignum_stall and rf_indirect_stall. See some lines above.

insn_dec_shared_i.wfi_insn: Originates out of the decoder similar like insn_dec_shared_i.subset which is already factored into other stall sources (the same examples as for insn_valid_i).

I added the wfi_resume flop exactly for this timing reason, to separate the core from the top. It delays the resume command by one cycle but that is totally fine.

etterli

Thanks @vogelpi for the review! Addressed your points.

etterli · 2026-07-17T06:45:12Z

+  // Interrupt is set when OTBN is no longer busy. This is the case when it gets paused or ends
+  // execution. The done interrupt is also set when a direct paused to locked transition occurs.
+  // This can happen under some faults where the secure wipe is skipped. These faults raise a fatal
+  // alert but for consistency we still set the interrupt. The faults are:
+  // - An invalid MuBi on lc_escalate_en_i
+  // - Spurious URND reseed ACK


This isn't actually a corner case, it is just to keep the behaviour the same as before. Without the WFI feature, when OTBN transitions from busy/executing to locked the DONE interrupt is still raised. See @h-filali question here. This new change is just to make the behaviour consistent and ensures that the DONE interrupt is also raised when transitioning from paused to locked.

I agree that the current description of the DONE interrupt (old and new) just says that it is raised when leaving a busy state. I added that it is also raised when OTBN locks itself. See updated otbn.hjson.

etterli · 2026-07-17T06:50:01Z

+  // wfi_pending signal has only effect if we haven't already received the resume command.
+  assign busy_execute_d = wfi_resume_d |
+                          ((busy_execute_q | start_d) &
+                           ~(done_core | (wfi_pending & ~wfi_resume_q)));


No I need the & ~wfi_resume_q part because wfi_pending is driven by the WFI instruction which is still executing when we resume, i.e., wfi_resume_q = 1. As long as the wfi instruction is executing, wfi_pending = 1 and thus busy_execute_d would be raised one cycle too late.

etterli · 2026-07-17T06:58:08Z

+  end
+  assign wfi_pending_o = wfi_pending_q;
+
+  assign stall = mem_stall | ispr_stall | rf_indirect_stall | mac_bignum_stall | wfi_stall;


Hm I don't think the new wfi_stall signal extends any path stall factors in at all. The wfi_stall is derived from:

wfi_resume_i: This originates directly from the flop wfi_resume_q in otbn.sv.

insn_valid_i: Is already factored in other stall sources like mac_bignum_stall and rf_indirect_stall. See some lines above.

insn_dec_shared_i.wfi_insn: Originates out of the decoder similar like insn_dec_shared_i.subset which is already factored into other stall sources (the same examples as for insn_valid_i).

I added the wfi_resume flop exactly for this timing reason, to separate the core from the top. It delays the resume command by one cycle but that is totally fine.

This specifies and documents the new WFI instruction for OTBN. The WFI instruction pauses the OTBN execution, unlocks the DMEM and let's Ibex change the DMEM content. Once the DMEM has been updated as desired, Ibex then can command the OTBN to resume the execution by issuing the RESUME command. Signed-off-by: Pascal Etterli <pascal.etterli@lowrisc.org>

This implements the WFI instruction in RTL for OTBN. Signed-off-by: Pascal Etterli <pascal.etterli@lowrisc.org>

etterli requested review from h-filali, nasahlpa and vogelpi July 3, 2026 15:21

etterli force-pushed the otbn-wfx branch from d9baf1c to cbaffd6 Compare July 3, 2026 15:31

etterli force-pushed the otbn-wfx branch 3 times, most recently from 2892dad to 9da6121 Compare July 14, 2026 13:40

etterli changed the title ~~[otbn] Synchronization interface~~ [otbn] Add synchronization interface between OTBN and Ibex Jul 14, 2026

etterli force-pushed the otbn-wfx branch from 9da6121 to a31e162 Compare July 14, 2026 13:54

etterli marked this pull request as ready for review July 14, 2026 13:55

etterli requested a review from a team as a code owner July 14, 2026 13:55

etterli requested review from rswarbrick and removed request for a team July 14, 2026 13:55

rswarbrick reviewed Jul 15, 2026

View reviewed changes

etterli force-pushed the otbn-wfx branch from a31e162 to 96a65af Compare July 15, 2026 13:49

etterli commented Jul 15, 2026

View reviewed changes

h-filali reviewed Jul 15, 2026

View reviewed changes

etterli force-pushed the otbn-wfx branch from 96a65af to e5e6ce2 Compare July 15, 2026 16:55

etterli force-pushed the otbn-wfx branch from e5e6ce2 to 5f7d411 Compare July 16, 2026 14:15

vogelpi approved these changes Jul 16, 2026

View reviewed changes

etterli force-pushed the otbn-wfx branch from 5f7d411 to bafc6d7 Compare July 17, 2026 07:00

etterli commented Jul 17, 2026

View reviewed changes

etterli added 2 commits July 17, 2026 10:31

[otbn,rtl] Implement WFI instruction

1063b6d

This implements the WFI instruction in RTL for OTBN. Signed-off-by: Pascal Etterli <pascal.etterli@lowrisc.org>

etterli force-pushed the otbn-wfx branch from bafc6d7 to 1063b6d Compare July 17, 2026 08:32

etterli added the CI:Rerun Rerun failed CI jobs label Jul 17, 2026

github-actions Bot removed the CI:Rerun Rerun failed CI jobs label Jul 17, 2026

etterli mentioned this pull request Jul 17, 2026

[otbn] Specify the URND control interface #30781

Draft


		// Interrupts ================================================================

		// Fires when OTBN gets paused or ends execution.

		// Mux core and bus access into IMEM.
		// While paused by a WFI insn the IMEM stays locked.

Uh oh!

Conversation

etterli commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etterli commented Jul 14, 2026

Uh oh!

rswarbrick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etterli left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etterli commented Jul 16, 2026

Uh oh!

vogelpi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etterli commented Jul 3, 2026 •

edited

Loading